Repository navigation
feat: SSM/STS, Lambda provided runtimes, and Redshift COPY on PostgreSQL for local data loads - #17
Merged
Merged
Conversation
SSM serves PutParameter, GetParameter (SecureString with WithDecryption), DeleteParameter, and DescribeParameters over AWS JSON 1.1, persisted to state.json. STS serves AssumeRole (inline session policy accepted, not evaluated) and GetCallerIdentity over the AWS Query protocol.
Default ports 18031 (ssm) and 18032 (sts). The dashboard lists SSM parameter metadata via DescribeParameters only, so values never reach it.
A provided* function's bootstrap runs as a local process against a per-environment Lambda Runtime API (invocation/next, response, error, init/error, chunked responses with error trailers). Environments are reused like interpreter runtimes. A missing or foreign-OS bootstrap fails with Runtime.InvalidEntrypoint. Extensions may register but get no events.
…-client sessions Each pgwire client now gets its own PostgreSQL connection, so temp tables, SETs, and transactions stay with that client and end on disconnect (they used to share one connection across clients). COPY ... FROM 's3://bucket/prefix' is loaded by devcloud on that session: text/DELIMITER, CSV, JSON 'auto', GZIP, IGNOREHEADER, NULL AS, EMPTYASNULL, BLANKSASNULL, with Redshift's empty-field NULL rules. Authorization options are ignored; unsupported options are refused. redshift-e2e covers a staged temp-table load in every format.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4e7b21a05a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Parse every object under the prefix before writing, then send all INSERTs as one simple Query so PostgreSQL applies them as a single implicit transaction (inside an explicit transaction a failure aborts it, as a failed COPY does on Redshift). Accept FROM after any whitespace, and detect COPY on bytes so a statement starting with multi-byte text cannot panic.
The ssm/sts blocks were inserted between cloudRun's enabled flag and its project/region, which then configured sts instead of Cloud Run.
SDKs sign the body's SHA-256 for these services without sending x-amz-content-sha256; treating its absence as UNSIGNED-PAYLOAD rejected correctly signed requests in strict mode.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
変更の目的
pushone-csa の data_load を devcloud だけでローカル実行するのに足りなかった機能を追加しました。SSM と STS を新しいサービスとして実装し、Lambda で
provided*ランタイム（Go の runner）を実行できるようにしました。また Redshift の PostgreSQL バックエンドで、COPY ... FROM 's3://...'を使えるようにしました。主な変更
PutParameter/GetParameter（SecureString とWithDecryption）/DeleteParameter/DescribeParametersを、AWS JSON 1.1 プロトコルで提供します。state.jsonに保存します。KMS は無いため、SecureString の値は平文で保存します。WithDecryptionを付けない取得には、平文ではなく不透明な値を返します。dashboard には一覧（値は出さない）を追加しました。AssumeRoleとGetCallerIdentityを AWS Query プロトコルで提供します。インラインの Policy は JSON として正しいかだけを確かめ、権限の評価はしません。状態は持ちません。provided*ランタイム:bootstrapをローカルのプロセスとして起動し、実行環境ごとに Lambda Runtime API（AWS_LAMBDA_RUNTIME_API）を立てます。対応しているのはinvocation/next・response・error・init/error、chunked 形式の本文と trailer です。実行環境は再利用され、ログは呼び出しごとに分割されます。bootstrapが見つからない、または別の OS 向けのバイナリの場合は、Runtime.InvalidEntrypointを返します。Extensions API は登録だけ受け付け、イベントは送りません。COPY ... FROM 's3://bucket/prefix'を devcloud 側で処理し、そのクライアントのセッション上で取り込みます。対応形式は text/DELIMITER、CSV、JSON 'auto'、GZIP、IGNOREHEADER、NULL AS、EMPTYASNULL、BLANKSASNULLです。空のフィールドを NULL にするかどうかは Redshift の規則に合わせました。対応していないオプションは無視せずエラーにします。検証
cargo test --workspacescripts/ssm-e2e.sh/scripts/sts-e2e.sh/scripts/lambda-e2e.sh/scripts/redshift-e2e.shnpm run typecheck/npm run build/npm test/npm run test:browser（Chromium）npm audit/cargo auditGetParameter(WithDecryption)で SecureString を取得できた。存在しないパラメータは*types.ParameterNotFoundとして判定できたlambda.StartWithOptions、darwin 向けにビルド）X-Amz-Function-Error: Unhandled）を確認。Linux 向けのバイナリはRuntime.InvalidEntrypointとビルドのヒントを返したsql.goの文をそのまま psql と pgx のPgConn().Execで実行）redshift_externalと cloudrun のdropped_instance_bounds...は、途中の実行で時々失敗しました。redshift_externalは main でも 8 回中 3 回失敗するため、この PR とは関係のない既存の不安定なテストです。最後の全体実行ではどちらも通っています。制約と、このPRで扱わないこと
sslmode=verify-fullで接続するため、そのままでは接続できません。ローカルでの接続方法は CSA 側で対応する前提です。strictモードのサービスでは受け付けられません（relaxedでは使えます）。